Foreach, future, and parallel computing with R




Bryan Lewis
Science




Map? Or is it map? hmmmm....
lapply → mclapply
Map → mcMap
clusterWhateverApply
"Real HPC programmers use MPI"
"Real HPC programmers use MPI"


Whatever.

Steve Weston

Package authors, code writers

Decide which parts of their code can run in parallel.

Package authors, code writers

Decide which parts of their code can run in parallel.

Users

Decide how to run in parallel.


> foreach == Philosophy
[1] TRUE

Haskell

Raskell

Despite the for-loop syntax, foreach works kinda like



Reduce(Map(...))
foreach(i = 1:2, j = 3:4, .combine = rbind) %dopar% { data.frame(i, j) }
foreach(i = 1:2, j = 3:4, .combine = rbind) %dopar% { data.frame(i, j) } i j 1 1 3 2 2 4 Warning message: executing %dopar% sequentially: no parallel backend registered
library(doParallel) registerDoParallel() foreach(i = 1:2, j = 3:4, .combine = rbind) %dopar% { data.frame(i, j) } i j 1 1 3 2 2 4
← THIS TIME, IN PARALLEL

The future package is not opinionated, you can do the same thing with any syntax!


Magic

Scoped objects are auto-magically exported

k <- pi foreach(i = 1:2, j = 3:4, .combine = rbind) %dopar% { data.frame(i, j, k) } i j k 1 1 3 3.141593 2 2 4 3.141593

Compose two foreach loops with the %:% operator


Or compose a foreach loop with a filter predicate using 'when'

foreach(i = 1:10, j = 1:10) %:% when(i < j) %dopar% { i + j }

Future works interoperably with foreach through doFuture, as do other parallel adapters

And a new guide to writing adapters:

https://github.com/bwlewis/writing_foreach_adapters/

Don't mourn...

Don't mourn...

parallelize!

1